My task is to do large scale inference via Sagemaker Batch Transform.

I have been following the tutorial: bring your own container, https://github.com/aws/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb

I have encountered many problems and solved them by searching stack overflow. However there is one problem that still causes the trouble.

When I run the same code and same dataset using 20 EC2 instances simultaneously, sometimes I get the error "Model container failed to respond to ping; Please ensure /ping endpoint is implemented and responds with an HTTP 200 status", and sometimes I don't.

What I find most frustrating is that, I have already do nothing for /ping (see code below)

@app.route("/ping", methods=["GET"])
def ping():
    """Determine if the container is working and healthy. In this sample container, we declare it healthy if we can load the model successfully."""
    # health = ScoringService.get_model() is not None  # You can insert a health check here
    # status = 200 if health else 404
    
    status = 200
    return flask.Response(response="\n", status=status, mimetype="text/csv")

How could the error still happen?

I read from some posts (e.g., How can I add a health check to a Sagemaker Endpoint? ) saying that "ping response should return within 2 seconds timeout".

How can I increase the ping response timeout? And in general, what can I do to prevent the error from happening?